Computers are used to perform a wide variety of applications in such diverse fields as finance, traditional and electronic commercial transactions, manufacturing, health care, telecommunications, etc. Most of these applications typically involve inputting or electronically receiving data, processing the data according to a computer program, then storing the results in a database, and perhaps transmitting the data to another application, messaging system, or client in a computer network. As computers become more powerful, faster, and more versatile, the amount of data that can be processed also increases.
Unfortunately, the raw data found in operational databases often exist as rows and columns of numbers and codes which, when viewed by individuals, appears bewildering and incomprehensible. Furthermore, the scope and vastness of the raw data stored in modern databases is overwhelming to a casual observer. Hence, applications were developed in an effort to help interpret, analyze, and compile the data so that it may be readily and easily understood by a human. This is accomplished by sifting, sorting, and summarizing the raw data before it is presented for display, storage, or transmission. Thereby, individuals can now interpret the data and make key decisions based thereon.
Extracting raw data from one or more operational databases and transforming it into useful information (e.g., data “warehouses” and data “marts”) is the function of analytic applications. In data warehouses and data marts, the data are structured to satisfy decision support roles rather than operational needs. A data warehouse utilizes a business model to combine and process operational data and make it available in a consistent way. Before the data are loaded into the data warehouse, the corresponding source data from an operational database are filtered to remove extraneous and erroneous records; cryptic and conflicting codes are resolved; raw data are translated into something more meaningful; and summary data that are useful for decision support, trend analysis and modeling or other end-user needs are pre-calculated. A data mart is similar to a data warehouse, except that it contains a subset of corporate data for a single aspect of business, such as finance, sales, inventory, or human resources.
In the end, the data warehouse or data mart is comprised of an “analytical” database containing extremely large amounts of data useful for direct decision support or for use in analytic applications capable of sophisticated statistical and logical analysis of the transformed operational raw data. With data warehouses and data marts, useful information is retained at the disposal of the decision makers and users of analytic applications and may be distributed to data warehouse servers in a networked system. Additionally, decision maker clients can retrieve analytical data resident on a remote data warehouse servers over a computer system network.
An example of the type of company that would use data warehousing is an online Internet bookseller having millions of customers located worldwide whose book preferences and purchases are tracked. By processing and warehousing these data, top executives of the bookseller can access the processed data from the data warehouse, which can be used for sophisticated analysis and to make key decisions on how to better serve the preferences of their customers throughout the world.
The rapid increase in the use of networking systems, including Wide Area Networks (WAN), the Worldwide Web and the Internet, provides the capability to transmit operational data into database applications and to share data contained in databases resident in disparate networked servers. For example, vast amounts of current transactional data are continuously generated by business-to-consumer and business-to-business electronic commerce conducted over the Internet. These transactional data are routinely captured and collected in an operational database for storage, processing, and distribution to databases in networked servers.
The expanding use of “messaging systems” and the like enhances the capacity of networks to transmit data and to provide interoperability between disparate database systems. Messaging systems are computer systems that allow logical elements of diverse applications to seamlessly link with one another. Messaging systems also provide for the delivery of data across a broad range of hardware and software platforms, and allow applications to interoperate across network links despite differences in underlying communications protocols, system architectures, operating systems, and database services. Messaging systems and the recent development of Internet access through wireless devices such as enabled cellular phones, two-way pagers, and hand-held personal computers, serve to augment the transmission and storage of data and the interoperability of disparate database systems.
In the current data warehouse/data mart networking environment, one general concern involves the sheer volume of data that must be dealt with. Often massive, multi-terabyte data files are stored in various server sites of data warehouses or in operational databases. Transmitting these massive amounts of data over WANs or the Internet is a troublesome task. The time needed to move the data is significant, and the probability that the data may contain an error introduced during transmission is increased. Also, the data are also vulnerable to interception by an unauthorized party. Furthermore, when the connection is lost in the process of transmitting the data over a network, there often is a need to retransmit large amounts of data already transmitted prior to the loss of connection, further increasing the time needed to move the data.
Accordingly, there is a need for a reliable, secure, authenticated, verifiable, and rapid system and/or method for the transmission of huge amounts of data, such as data in a data warehouse/mart, over networks such as WANs and the Internet. The present invention provides a novel solution to this need.